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METHOD AND SYSTEM FOR LUMBVANCE NOISE FILTERING 



Cross-Reference to Related Applicationfs^ 

The present application claims priority under 35 U.S.C. §1 19 to Korean Patent 
5 Application No. 2003-0037268, filed on June 10, 2003, which is incorporated herein by 
reference in its entirety. 

Technical Field 

The present invention relates generally to image pick-up devices, and more 
10 particularly, to luminance noise filtering for a pixel location using a region of pixel data 
directly from an image sensor such that line memory capacity is minimized. 

Background of the Invention 

Fig. 1 illustrates an image pick-up device 102 such as a camera system that 
15 includes an image sensor 104. The image sensor 104 generates pixel data for an image of 
an object 106 that is projected through an objective lens 108 onto the image sensor 104. 
For example, the image sensor 104 may be a CIS (CMOS image sensor) commonly used 
in hand-held devices such as cell phones and PDA's (personal digital assistants). 

A signal processor 110 manipulates the pixel data from the image sensor 104 for 
20 showing the image of the object 106 on a display 1 12, or for ftirther processing by an 
image recognition system 1 14, or for sending the image via a transmission system 116 
such that the image is shown on a remote display 118. Referring to Figs. 1 and 2, the 
image sensor 104 generates pixel data 120 according to a Bayer filter array overlying the 
image sensor 104. 

25 With the Bayer filter array, the image sensor 104 generates an intensity signal of a 

respective color at each pixel location. A square labeled with an "R" is for a pixel 
location on the image sensor 104 that generates an intensity signal of the red color 
component. Similarly, a square labeled with a "G" is for a pixel location on the image 
sensor 104 that generates an intensity signal of the green color component. Further, a 

30 square labeled with a "B" is for a pixel location on the image sensor 104 that generates an 
intensity signal of the blue color component. 

An interpolation algorithm is then used by the signal processor 1 10 to determine 

1 
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the full set of intensity signals of the respective interpolated RGB color components for 
each of the pixel locations. The interpolation algorithm uses the pixel data of the Bayer 
color filter array 120 for such a determination. 

Such an interpolation algorithm is known to one of ordinary skill in the art as 
5 disclosed in U.S. Patent No. 5,382,976, U.S. Patent No. 5,506,619, or U.S. Patent No. 
6,091,862. For determining the interpolated color components R\ G\ and B' at a 
particular pixel location 124 with such an interpolation algorithm, a region of pixel data 
126 surrounding that pixel location 124 is used as illustrated in Fig. 3. 

Temporal noise affects the quality of the image of the object as detected and 

10 generated by the image pick-up device 102. Temporal noise is the variation in the output 
from the image sensor 104 even under uniform illumination onto the image sensor 104. 
Such temporal noise may arise from shot noise and 1/f noise at the photo-diodes of the 
image sensor 104, from thermal noise at the transistors and other circuit components used 
within the image sensor 1 04, or from quantization error of an A/D (analog to digital) 

15 converter used within the image sensor 104. 

Such temporal noise increases with brightness of the image. However, the 
detrimental effect of the temporal noise on the image is greater at lower illumination 
because the SNR (signal to noise ratio) decreases with lower illumination. In fact, 
temporal noise sets a limit on the dynamic range of the image sensor 104 under dark 

20 conditions. 

Fig. 4 illustrates a prior art process for reducing the effect of such temporal noise. 
The pixel data 120 is generated with the Bayer color filter array at the image sensor 104. 
The signal processor 110 interpolates such pixel data 120 to generate the respective 
interpolated RGB color components 122 A, 122B, and 122C that are stored within a frame 

25 memory device 122 of the prior art. 

In the prior art, after the interpolated RGB color components 122 A, 122B, and 
122C for an nxn array of pixel locations are generated and stored in the frame memory 
device 122, a noise reducing block 132 uses such interpolated RGB color components for 
reducing the deleterious effects of the temporal noise. Fig. 4 shows the noise reducing 

30 block 132 using the 3x3 arrays of the interpolated RGB color components 122 A, 122B, 
and 122C. However, other prior art noise reducing processes may also use 5x5, 7x7, or 
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other nxn arrays of the interpolated RGB color components. 

In any case for the prior art noise reducing process, the capacity of the frame 
memory device 122 is sufficient to store the nxn arrays of interpolated RGB color 
components used by the noise reducing block 132. However, such a relatively large 
5 capacity of the frame memory device 122 is disadvantageous when the camera system 
102 is incorporated as part of a hand-held device such as a cell phone or a PDA for 
example. Thus, elimination of the frame memory device 122 is desired for a smaller 
device size, lower power dissipation, and lower cost especially when the camera system 
102 is incorporated into a hand-held device. 

10 

Summary of the Invention 

Accordingly, in a general aspect of the present invention, luminance noise filtering 
is performed for a pixel location using a relatively small region of pixel data directly from 
the image sensor such that the frame memory may be eliminated. 
15 In a general embodiment of the present invention, in a method and system for 

luminance noise filtering, a region of pixel data from the image sensor is used for 
determining a virtually filtered luminance for a pixel location within the region. In an 
example embodiment, the virtually filtered luminance is determined by averaging the 
respective pixel data multiplied with a respective weighting coefficient for each pixel 
20 location of the region. 

In another embodiment of the present invention, the color components for the 
pixel location are determined from the region of pixel data. 

In a fiirther embodiment of the present invention, a reference luminance is 
determined for the pixel location from the color components. A final luminance of the 
25 pixel location is selected between the virtually filtered luminance and the reference 

luminance depending on an adaptive luminance that indicates the brightness of the image. 
The present invention provides adaptive noise filtering by selecting the virtually filtered 
luminance as the final luminance for a darker image. 

In this manner, noise filtering is performed for a pixel location within a region of 
30 pixel data using virtual luminance that is determined using such a region of pixel data 

directly from the image sensor. Thus, frame memory for storing interpolated pixel data is 
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eliminated with the present invention. The elimination of frame memory is especially 
advantageous for smaller device size, lower power dissipation, and lower cost of the 
camera system incorporated into a hand-held device. 

These and other features and advantages of the present invention will be better 
5 understood by considering the following detailed description of the invention which is 
presented with the attached drawings. 

Brief Description of the Drawing s 

Fig. 1 shows components of an image pick-up device such as a camera system, 
1 0 according to the prior art; 

Fig. 2 illustrates interpolation of pixel data generated with a Bayer color filter 
array into color components, according to the prior art; 

Fig. 3 illustrates use of a region of pixel data generated with a Bayer color filter 
array for determining the color components of a pixel location within the region, 
1 5 according to the prior art; 

Fig. 4 illustrates a process for performing noise reduction after a nxn array of color 
components has been determined, according to the prior art; 

Fig. 5 illustrates a process for perfonning noise filtering using a region of pixel 
data for determining the luminance and the color components of a pixel location, 
20 according to an example embodiment of the present invention; 

Fig. 6 shows a block diagram of a system for performing noise filtering as 
illustrated in Fig. 5 such that frame memory is eliminated, according to an example 
embodiment of the present invention; 

Fig. 7 shows a flowchart during operation of the system of Fig. 6, according to an 
25 example embodiment of flie present invention; 

Fig. 8 illustrates use of a region of pixel data generated with the Bayer color filter 
array for determining color components of a pixel location within the region in a step of 
the flowchart of Fig. 7, according to an example embodiment of the present invention; 
Figs. 9A, 9B, and 9C illustrate use of the region of pixel data of Fig. 8 for 
30 determining a plurality of virtual luminance arrays, according to an example embodiment 
of the present invention; 
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Fig. 10 shows a block diagram of a noise filter within the system of Fig. 6, 
according to an example embodiment of the present invention; 

Fig. 1 1 shows an example graph used by a data processor within the system of Fig. 
6 for determining a threshold value depending on an adaptive luminance, according to an 
5 example embodiment of the present invention; and 

Fig. 12 shows another example graph used by the data processor within the system 
of Fig. 6 for determining the threshold value depending on an auto exposure gain of the 
camera system, according to an example embodiment of the present invention. 

The figures referred to herein are drawn for clarity of illustration and are not 
10 necessarily drawn to scale. Elements having the same reference number in Figs. 1, 2, 3, 
4, 5, 6, 7, 8, 9, 10, 1 1, and 12 refer to elements having similar structure and fimction. 



Detailed Description 

Referring to Figs. 5 and 6, a system 200 of a general aspect of the present 
15 invention performs noise filtering using a region of pixel data directly fi-om the image 
sensor 104. In addition, the noise filtering is performed during determination of the 
limiinance using such a region of pixel data for a single pixel location within the region. 
Fig. 7 shows a flowchart of steps during operation of the system 200 of Fig. 6. 

The system 200 is typically implemented within the signal processor 1 10 for 
20 manipulating pixel data fi-om the image sensor 104. Referring to Figs. 6, 7, and 8, a RGB 
matrix 202 includes a line memory device 204 for inputting and storing a region of pixel 
data 206 generated with a Bayer color filter array at the image sensor 104 (step 302 of 
Fig. 7). 

The line memory device 204 may be implemented with any type of data storage 
25 devices. For example, the line memory device 204 may store pixel data for 4xH pixel 
locations with H being the number of columns of pixels at the image sensor 104 (such as 
648 columns for example) when the image sensor 1 04 outputs pixel data row by row. 

The RGB matrix 202 also includes an interpolation processor 208 for determining 
interpolated color components R', B', and G' of a pixel location 210 within the region 
30 206 (step 304 of Fig. 7). Referring to Fig. 8, the interpolation processor 208 interpolates 
the region of pixel data 206 to generate the interpolated color components R', B\ and G'. 
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Such interpolation algorithms for generating the interpolated color components R', B% 
and G' are known to one of ordinary skill in the art. 

In addition, the RGB matrix 202 determines a reference luminance, YIH, from the 
interpolated color components R', B', and G' as follows (step 304 of Fig. 7): 
5 YIH = (19*R' + 38*G' + 7*B')/64 

Such a reference luminance YIH is calculated according to a conventional standard in the 
industry for calculating luminance as known to one of ordinary skill in the art. 

Furthermore, the RGB matrix 202 includes a virtual luminance processor 212 for 
determining virtual luminance arrays YOV, YIV, and Y2V (step 304 of Fig. 7). Referring 
10 to Fig. 9A, a first virtual luminance array YOV = [ A B C], with A, B, and C each being a 
respective average of the intensity values of a respective four pixels included in a box 
correspondingly labeled as A, B, and C in the region of pixel data 206. 

Similarly, referring to Fig. 9B, a second virtual luminance array Yl V = [D E F], 
with D, E, and F each being a respective average of the intensity values of a respective 
15 four pixels included in a box correspondingly labeled as D, E, and F in the region of pixel 
data 206. Also, referring to Fig. 9C, a third virtual luminance array Y2V = [G H I], with 
G, H, and I each being a respective average of the intensity values of a respective four 
pixels included in a box correspondingly labeled as G, H, and I in the region of pixel data 
206. In this manner, the virtual luminance arrays YOV, YIV, and Y2V are determined for 
20 the pixel location 210 using the region of pixel data 206 that also determines the 
interpolated color components R', G', and B' for the pixel location 210. 

Referring back to Fig. 6, the RGB matrix 202 sends the reference luminance YIH 
and the virtual luminance arrays YOV, YIV, and Y2V to a noise filter 214. In addition, 
the RGB matrix 202 sends the interpolated color components R', G', and B' to a 
25 chrominance signal processor 215 within a Y/C (luminance/chrominance) processor 216. 
Furthermore, the RGB matrix 202 sends the reference limiinance YIH and the 
interpolated color components R', G', and B' to a data processor 218. 

Fig. 10 shows a block diagram for an example embodiment of the noise filter 214 
that determines a virtually filtered luminance GOUT (step 306 of Fig. 7) fi-om the virtual 
30 luminance arrays YOV, YIV, and Y2V. The noise filter 214 includes multipliers 222, 
224, and 226 and an adder 228 within a multiplying and summing block 230. 
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The first multiplier 222 multiplies the first luminance array YOV with a first 
weighted coefficient array GADO [39 63 39] as follows: 

39*A + 63*B + 39*C 
Similarly, the second multiplier 224 multiplies the second luminance array Yl V with a 
5 second weighted coefficient array GADl [63 104 63] as follows: 

63*D+ 104*E + 63*F. 
Also, the third multiplier 226 multiplies the third luminance array Y2V with a third 
weighted coefficient array GAD2 [39 63 39] as follows: 

39*G + 63*H + 39*L 

10 The adder 228 sums together the resulting values fi-om the multipliers 222, 224, 

and 226. A shifter 232 within a brightness control block 234 divides the result fi-om the 
adder 228 with a sum of all the coefficients of GADO, GADl, and GAD2 (i.e., 512). A 
fourth multiplier 236 within the brightness control block 234 multiplies the result fi-om 
the shifter 232 by a luminance compensation factor a, to generate the virtually filtered 

1 5 luminance GOUT. 

In one embodiment of the present invention, the luminance compensation factor a 
and the weighted coefficient arrays GADO, GADl, and GAD2 are determined using a 
Gaussian distribution equation for optimum image quality and are stored within a data 
register 220 in Fig. 6. In this manner, the virtually filtered luminance GOUT is 

20 determined by averaging a respective pixel data multiplied with a respective weighting 
coefficient for each pixel location of the region 206. 

In addition, referring to Fig. 6, the data processor 218 determines a threshold value 
THV (step 306 of Fig. 7). Referring to Fig. 1 1, the data processor 218 determines THV 
dependent on an adaptive luminance, AY, as illustrated by the graph 223 of Fig. 11. The 

25 adaptive luminance, AY, indicates the overall brightness of a previous image, in one 
example embodiment of the present invention. 

For example, in Fig. 12, THV is determined according to another graph 225 
dependent on the auto exposure gain used by the camera system 102. The auto exposure 
gain indicates a brightness of the image to be detected by the image sensor 102 within the 

30 camera system 102, as known to one of ordinary skill in the art. A higher auto exposure 
gain indicates a darker image for a higher THV according to the graph 225 of Fig. 12. In 



7 



Case No. 04-02 

such an example, the auto exposure gain indicates the adaptive luminance AY. 

In another embodiment of the present invention, the reference luminance YIH is 
used as indicating the adaptive luminance AY in Fig. 1 1 . In that case, THV is determined 
depending on a brightness level for each pixel location. Alternatively, the adaptive 
5 luminance AY is an average of the respective reference luminance YIH for a 

predetermined region of pixel data. In any case, the THV is determined to be higher for a 
darker image according to the graph 223 of Fig. 1 1 . 

Referring back to Fig. 10, a comparing block 238 includes a subtractor 240 and an 
absolute value generator 242 that determine an absolute of the difference between the 
10 virtually filtered luminance GOUT and the reference luminance YIH (i.e., delta) (step 
308 of Fig. 7). A comparator 244 within the comparing block 238 compares delta with 
THV (step 310 in Fig. 7). 

In Fig. 10, a multiplexer 246 inputs the reference luminance YIH and the virtually 
filtered luminance GOUT. The comparator 244 controls the multiplexer 246 to select and 
15 output the virtually filtered luminance GOUT as a final luminance of the pixel location 
210 when delta is less than or equal to THV (step 312 of Fig. 7). Alternatively, the 
comparator 244 controls the multiplexer 246 to select and ou^ut the reference luminance 
YIH as the final luminance of the pixel location 210 when delta is greater than THV (step 
314 of Fig. 7). 

20 If the pixel location 210 is a last pixel location for an image (step 316 of Fig. 7), 

the flowchart of Fig. 7 ends. Otherwise, steps 302, 304, 306, 308, 310, 312, 314, and 316 
are repeated for another pixel location with another region of pixel data fi-om the image 
sensor 104. Such steps are repeated for determining the respective luminance and 
interpolated color components for each pixel location with a respective region of pixel 

25 data of the image. 

Generally, a larger array of pixel data is generated fi-om the image sensor 104 for 
producing a smaller array of the processed image, as known to one of ordinary skill in the 
art. Pixel data fi^om locations toward the outer perimeter of the image sensor 104 are used 
for image signal processing of adjacent pixel locations toward the center of the processed 

30 image. However, such pixel locations toward the outer perimeter become cut off from the 
processed image because a sufficient region of pixel data surrounding such a pixel 
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location is not available, as known to one of ordinary skill in the art. Steps , 304, 306, 
308, 310, 312, 314, and 316 of Fig. 7 are repeated for each of the pixel locations for the 
processed image. 

In this manner, noise filtering is performed for a single pixel location using a 
5 region of pixel data 206 directly from the image sensor 104 while determining the final 
luminance YIH/GOUT for the pixel location 210. The region of pixel data 206 is also 
used for determining the interpolated color components R', G', and B' of the pixel 
location 210. Because noise filtering is performed by using the region of pixel data 206 
directly from tiie image sensor 104, frame memory for storing interpolated pixel data may 

10 be eliminated with the present invention. 

Thus, the capacity of the memory device included in the image pick-up device 
may be minimized since the luminance noise filter uses the image data directly from the 
image sensor for determining the final luminance YIH/GOUT and the interpolated color 
components R', G', and B' of the pixel location 210. Such smaller memory capacity is 

15 advantageous for smaller device size, lower power dissipation, and lower cost especially 
when the camera system is incorporated into a hand-held device. 

In addition, the present invention provides adaptive noise filtering by varying the 
threshold value THV depending on the brightness of the image. For a brighter image, the 
reference luminance YIH is selected as the final Ituninance instead of the virtually filtered 

20 luminance GOUT. Noise filtering introduces distortion to the image, and the effect of 
temporal noise is less for a brighter image. Thus, the reference luminance YIH without 
distortion from virtual noise filtering is selected as the final luminance for a brighter 
image. On the other hand, the deleterious effect of temporal noise is greater for a darker 
image. Thus, the virtually filtered luminance GOUT with noise filtering is selected as the 

25 final luminance for a darker image. 

Referring back to Fig. 6, a luminance signal processor 217 within the Y/C 
processor 216 processes the final luminance YIH/GOUT for performing contour 
compensation to output a luminance signal Y for a Y/C (luminance/chrominance) 
formatter 219. A chrominance signal processor 217 within the Y/C processor 216 

30 receives the interpolated color components R*, G', and B' and auto white balance data 
(AWBD) from the data processor 218 to generate fiirther color components R"-Y and 
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B"-Y for the Y/C formatter 219. The R" is an intensity of the interpolated color 
component R' which is adjusted by the auto white balance data, and the B" is an intensity 
of the interpolated color component B' which is adjusted by the auto white balance data. 

The Y/C formatter 219 generates luminance and chrominance data Y, Cb, Cr/R, 
and color data R"% and B'" according to a standard as required by the display 1 12, 
the image recognition system 1 16, or the transmission system 116. Such components of 
the Y/C processor 216 and the Y/C formatter 219 are known to one of ordinary skill in the 
art. The data processor 218 also determines and outputs to the image pick-up device auto 
exposure control data AED from YIH, R', G', and B*, as known to one of ordinary skill 
in the art. The present invention lies in the darkly outlined components of the RGB 
matrix 202, the noise filter 214, the data processor 218, and the data register 220 in Fig. 6, 
and the other components of Fig. 6 are known to one of ordinary skill in the art. 

The foregoing is by way of example only and is not intended to be limiting. For 
example, the present invention is described for the camera system 102 that may be part of 
a hand-held device. However, the present invention may be used for any type of imaging 
device performing image signal processing. In addition, the components illustrated and 
described herein for an example embodiment of the present invention may be 
implemented with any combination of hardware and/or software and in discrete and/or 
integrated circuits. In addition, any number as illustrated and described herein is by way 
of example only. For example, any number of pixels as illustrated and described herein is 
by way of example only. 

The present invention is limited only as defined in the following claims and 
equivalents thereof. 
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